Night Shift: fix full table scan in word details query#24
Night Shift: fix full table scan in word details query#24
Conversation
The getWordDetailsWithSongs query was loading ALL words from the database and filtering in memory. As the words table grows, this becomes a major performance bottleneck — called every time a user opens the word details modal. Now uses a by_persian index for O(log n) lookup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughThe changes add a new database index on the Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
PR Compliance Guide 🔍Below is a summary of compliance checks for this PR:
Compliance status legend🟢 - Fully Compliant🟡 - Partial Compliant 🔴 - Not Compliant ⚪ - Requires Further Human Verification 🏷️ - Compliance label |
||||||||||||||||||||||||
PR Code Suggestions ✨Explore these optional code suggestions:
|
||||||||||||
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
convex/wordProgress.ts (1)
6-17: 🛠️ Refactor suggestion | 🟠 MajorMove modified Convex queries/mutations to
songs.ts(or update the guideline).The changed query/mutation implementations are in
convex/wordProgress.ts, but the repository guideline requires queries/mutations insongs.ts.As per coding guidelines, “Use Convex schema.ts for database schema definitions and songs.ts for queries and mutations.”
Also applies to: 20-32, 35-47, 51-71, 74-92, 96-129, 133-166, 169-216, 219-260, 263-357, 362-467, 469-532, 535-574, 577-605
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@convex/wordProgress.ts` around lines 6 - 17, The Convex query implementations (e.g., getByUser in convex/wordProgress.ts and the other queries/mutations listed) must be moved into songs.ts to follow the project guideline; open songs.ts and copy each query/mutation implementation (keeping function names like getByUser) into that file, export them from songs.ts, update any imports that referenced convex/wordProgress.ts to point to songs.ts, and either delete convex/wordProgress.ts or leave a thin forwarding stub that re-exports the same names from songs.ts to avoid breaking references; ensure Convex query/mutation registrations remain intact and run tests/TypeScript build to confirm no unresolved imports.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@convex/wordProgress.ts`:
- Around line 61-65: The queries against the "wordProgress" table use
.withIndex("by_user").filter(...) which forces a post-index scan; add composite
indexes named by_user_word (userId + word) and by_user_persian (userId +
persian) to the Convex schema and replace the .withIndex("by_user") calls with
.withIndex("by_user_word") or .withIndex("by_user_persian") at the affected call
sites (e.g., the query in wordProgress.ts that calls
ctx.db.query("wordProgress").withIndex("by_user", (q) => q.eq("userId",
userId)).filter((q) => q.eq(q.field("persian"), persian)).first()) so each query
can match the composite index instead of filtering after the index; update all
listed locations (the other withIndex("by_user").filter usages) to pick the
correct composite index for their filter key.
---
Outside diff comments:
In `@convex/wordProgress.ts`:
- Around line 6-17: The Convex query implementations (e.g., getByUser in
convex/wordProgress.ts and the other queries/mutations listed) must be moved
into songs.ts to follow the project guideline; open songs.ts and copy each
query/mutation implementation (keeping function names like getByUser) into that
file, export them from songs.ts, update any imports that referenced
convex/wordProgress.ts to point to songs.ts, and either delete
convex/wordProgress.ts or leave a thin forwarding stub that re-exports the same
names from songs.ts to avoid breaking references; ensure Convex query/mutation
registrations remain intact and run tests/TypeScript build to confirm no
unresolved imports.
ℹ️ Review info
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (2)
convex/schema.tsconvex/wordProgress.ts
📜 Review details
🧰 Additional context used
📓 Path-based instructions (5)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use TanStack Start framework with Bun runtime for the application
Tests must pass locally via
bun run testbefore committing code, as Husky pre-commit hooks will block commits with failing tests
Files:
convex/schema.tsconvex/wordProgress.ts
convex/schema.ts
📄 CodeRabbit inference engine (AGENTS.md)
Define database schema in convex/schema.ts using Convex
Files:
convex/schema.ts
convex/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
Use Convex for database queries, mutations, and authentication configuration
Files:
convex/schema.tsconvex/wordProgress.ts
**/*.ts
📄 CodeRabbit inference engine (CLAUDE.md)
All new helpers and utilities MUST have corresponding test files
Files:
convex/schema.tsconvex/wordProgress.ts
convex/**/*.ts
📄 CodeRabbit inference engine (CLAUDE.md)
convex/**/*.ts: NEVER create .js files in the convex/ folder - only .ts files belong there
Before starting Convex dev server, always runrm -f convex/*.jsto clean compiled JavaScript files
Use Convex schema.ts for database schema definitions and songs.ts for queries and mutations
Files:
convex/schema.tsconvex/wordProgress.ts
🧠 Learnings (4)
📚 Learning: 2026-01-28T12:17:53.877Z
Learnt from: CR
Repo: EtanHey/songscript PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-01-28T12:17:53.877Z
Learning: Applies to convex/**/*.ts : Use Convex schema.ts for database schema definitions and songs.ts for queries and mutations
Applied to files:
convex/schema.tsconvex/wordProgress.ts
📚 Learning: 2026-01-23T18:12:38.519Z
Learnt from: CR
Repo: EtanHey/songscript PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-01-23T18:12:38.519Z
Learning: Applies to convex/schema.ts : Define database schema in convex/schema.ts using Convex
Applied to files:
convex/schema.ts
📚 Learning: 2026-01-23T18:12:49.193Z
Learnt from: CR
Repo: EtanHey/songscript PR: 0
File: prd-json/AGENTS.md:0-0
Timestamp: 2026-01-23T18:12:49.193Z
Learning: Applies to prd-json/**/convex/**/*{score,progress,leaderboard}*.{ts,tsx,js} : Use Progress Score Formula: (words_learned × multiplier) + (lines_completed × multiplier × 0.5)
Applied to files:
convex/wordProgress.ts
📚 Learning: 2026-01-23T18:12:38.519Z
Learnt from: CR
Repo: EtanHey/songscript PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-01-23T18:12:38.519Z
Learning: Applies to convex/*.ts : Use Convex for database queries, mutations, and authentication configuration
Applied to files:
convex/wordProgress.ts
🔇 Additional comments (2)
convex/schema.ts (1)
51-53: Nice optimization:by_persianindex is the right fix for this lookup path.This directly removes the prior scan-heavy pattern for Persian word lookups in modal flows.
convex/wordProgress.ts (1)
473-476: Great improvement on word-details lookup path.Switching to
words.by_persiandirectly addresses the full-scan bottleneck in this modal query.
| const progress = await ctx.db | ||
| .query("wordProgress") | ||
| .withIndex("by_user", (q) => | ||
| q.eq("userId", userId) | ||
| ) | ||
| .withIndex("by_user", (q) => q.eq("userId", userId)) | ||
| .filter((q) => q.eq(q.field("persian"), persian)) | ||
| .first(); |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Check current indexes in convex/schema.ts for wordProgress
rg -n 'wordProgress.*defineTable|\.index\(' convex/schema.ts -A 40 | head -100Repository: EtanHey/songscript
Length of output: 4583
🏁 Script executed:
# Check the specific patterns at the mentioned lines in convex/wordProgress.ts
rg -n '\.withIndex\("by_user"' convex/wordProgress.ts -C 3Repository: EtanHey/songscript
Length of output: 3042
🏁 Script executed:
# Check if composite indexes are already defined
rg -n '\.index\("by_user' convex/schema.ts -A 2Repository: EtanHey/songscript
Length of output: 1033
🏁 Script executed:
# Verify the coding guideline - check if there are queries/mutations in wordProgress.ts vs songs.ts
echo "=== Exports in convex/wordProgress.ts ===" && \
rg -n '^export (const|function|default)' convex/wordProgress.ts | head -20 && \
echo "" && \
echo "=== Exports in convex/songs.ts ===" && \
rg -n '^export (const|function|default)' convex/songs.ts | head -20Repository: EtanHey/songscript
Length of output: 1092
Add composite indexes by_user_word and by_user_persian to eliminate post-index filtering scans.
The by_user index is single-column only. All flagged call sites use .withIndex("by_user").filter(...) patterns, which scan all user rows before applying the filter—causing O(user_rows) performance degradation in hot paths. The codebase already defines composite indexes for visitorId-based lookups (by_visitor_word, by_visitor_persian); apply the same pattern for userId.
Required schema changes
# convex/schema.ts (wordProgress indexes)
.index("by_user", ["userId"])
+.index("by_user_word", ["userId", "wordId"])
+.index("by_user_persian", ["userId", "persian"]),# convex/wordProgress.ts examples
- .withIndex("by_user", (q) => q.eq("userId", userId))
- .filter((q) => q.eq(q.field("wordId"), wordId))
+ .withIndex("by_user_word", (q) =>
+ q.eq("userId", userId).eq("wordId", wordId)
+ )
- .withIndex("by_user", (q) => q.eq("userId", userId))
- .filter((q) => q.eq(q.field("persian"), persian))
+ .withIndex("by_user_persian", (q) =>
+ q.eq("userId", userId).eq("persian", persian)
+ )Affected locations: lines 28–30, 43–45, 63–65, 84–86, 104–106, 141–143, 177–179, 227–229, 552–554.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@convex/wordProgress.ts` around lines 61 - 65, The queries against the
"wordProgress" table use .withIndex("by_user").filter(...) which forces a
post-index scan; add composite indexes named by_user_word (userId + word) and
by_user_persian (userId + persian) to the Convex schema and replace the
.withIndex("by_user") calls with .withIndex("by_user_word") or
.withIndex("by_user_persian") at the affected call sites (e.g., the query in
wordProgress.ts that calls ctx.db.query("wordProgress").withIndex("by_user", (q)
=> q.eq("userId", userId)).filter((q) => q.eq(q.field("persian"),
persian)).first()) so each query can match the composite index instead of
filtering after the index; update all listed locations (the other
withIndex("by_user").filter usages) to pick the correct composite index for
their filter key.
User description
Automated improvement by Golems Night Shift.
fix full table scan in word details query
PR Type
Enhancement, Bug fix
Description
Add
by_persianindex to words table for O(log n) lookupFix full table scan in
getWordDetailsWithSongsqueryRefactor code formatting for consistency across wordProgress queries
Diagram Walkthrough
File Walkthrough
schema.ts
Add by_persian index to words tableconvex/schema.ts
by_persianindex to the words table schemawordProgress.ts
Fix word details query and format code consistentlyconvex/wordProgress.ts
getWordDetailsWithSongsquery to useby_persianindex instead offull table scan
ctx.db.query("words").collect()with indexed query usingwithIndex("by_persian")consistency
Summary by CodeRabbit
Bug Fixes
Refactor